[HUDI-6509] Add GitHub CI for Java 17 #9136

CTTY · 2023-07-06T20:41:29Z

Change Logs

Add a new GitHub action to test Hudi spark against Java 17. Some known failures would be excluded as they failed due to Hadoop dependencies like MiniDFSCluster

Impact

None

Risk level (write none, low medium or high below)

None

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

CTTY · 2023-07-12T22:28:54Z

@hudi-bot run azure

CTTY · 2023-07-18T16:58:24Z

It seems validate-bundles(flink1.17, Spark 3.4, Spark 3.4.0) just consistently fail with JDK17 on issue below:

Connecting to jdbc:hive2://localhost:10000/default
23/07/17 17:45:48 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Cannot run commands specified using -e. No current connection
Error: validate.sh HiveQL validation failed.
Error: Process completed with exit code 1.

Need to look into this, otherwise everything looks good. newly added docker-test-java17 and test-spark-java17 are working fine

.github/workflows/bot.yml

yihua · 2023-07-18T17:01:42Z

.github/workflows/bot.yml

+        run:
+          mvn test -Pfunctional-tests -Pjava17 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -pl "$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS
+
+  docker-test-java17:


Could this be run with validate-bundles since it already validates bundles on Java 17? Any reason to have a separate job here?

those tests are generally the same as bundle validation but there is one difference: It requires building Hudi within Docker as it runs tests with mvn test command, while bundle validation build Hudi outside of Docker and only copy jars/bundles to Docker. If we consolidates them as one then the job would need to build twice, it would make the job much slower.

If we just seperate them into 2 jobs then docker test can only build hudi-common modules in Docker which is relatively fast and bundle validation can keep the same behavior

I think it's better to have one job and docker setup so it's easier to maintain. We can fix the repeated mvn build later on.

hudi-common/pom.xml

hudi-common/src/test/java/org/apache/hudi/avro/TestHoodieAvroUtils.java

hudi-common/src/test/java/org/apache/hudi/common/fs/TestHoodieWrapperFileSystem.java

hudi-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormat.java

...common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormatAppendFailure.java

packaging/bundle-validation/Dockerfile

packaging/bundle-validation/ci_run.sh

yihua · 2023-07-18T20:12:49Z

packaging/bundle-validation/docker_test_java17.sh

+#    - <dataset name>/schema.avsc
+#    - <dataset name>/data/<data files>
+#################################################################################################
+


Could we consolidate the test logic of this into validate.sh and reuse existing validate-bundle job?

yihua · 2023-07-18T20:15:53Z

pom.xml

    <flink.connector.kafka.artifactId>flink-connector-kafka</flink.connector.kafka.artifactId>
    <flink.hadoop.compatibility.artifactId>flink-hadoop-compatibility_2.12</flink.hadoop.compatibility.artifactId>
-    <rocksdbjni.version>5.17.2</rocksdbjni.version>
+    <rocksdbjni.version>7.5.3</rocksdbjni.version>


does RocksDB 5.17.2 not work? Dependency version upgrade has larger impact.

Right, RocksDB 5.17.2 would throw NoClassDefException when running TestHoodieLogFormat

[ERROR] testBasicAppendAndScanMultipleFiles{DiskMapType, boolean, boolean, boolean}[10] Time elapsed: 0.118 s <<< ERROR! 2023-07-13T23:41:36.1420947Z java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.DBOptions

This seems fishy. Does this only happen for Java 17?

Yes, for Java 8 it works fine. This issue only pops up for Java 17

After reading the code, RocksDbBasedFileSystemView is the only place using RocksDb. So this should be ok.

pom.xml

style/checkstyle.xml

hudi-bot · 2023-07-24T21:51:26Z

CI report:

a0e7207 UNKNOWN
cda1e77 UNKNOWN
6b33d37 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

This reverts commit 68f6b69.

yihua

LGTM. Thanks for adding validations on Java 17!

yihua · 2023-07-25T18:42:00Z

.github/workflows/bot.yml

+          SCALA_PROFILE: ${{ matrix.scalaProfile }}
+          SPARK_PROFILE: ${{ matrix.sparkProfile }}
+          SPARK_MODULES: ${{ matrix.sparkModules }}
+        if: ${{ !endsWith(env.SPARK_PROFILE, '3.2') }} # skip test spark 3.2 as it's covered by Azure CI


nit: not required.

yihua · 2023-07-25T18:42:05Z

.github/workflows/bot.yml

+          SCALA_PROFILE: ${{ matrix.scalaProfile }}
+          SPARK_PROFILE: ${{ matrix.sparkProfile }}
+          SPARK_MODULES: ${{ matrix.sparkModules }}
+        if: ${{ !endsWith(env.SPARK_PROFILE, '3.2') }} # skip test spark 3.2 as it's covered by Azure CI


nit: not required.

yihua · 2023-07-25T18:42:13Z

.github/workflows/bot.yml

+          SPARK_PROFILE: ${{ matrix.sparkProfile }}
+          SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
+          SCALA_PROFILE: 'scala-2.12'
+        if: ${{ env.SPARK_PROFILE >= 'spark3.4' }} # Only support Spark 3.4 for now


nit: not required.

hudi-common/pom.xml

yihua · 2023-07-25T18:53:38Z

packaging/bundle-validation/run_docker_java17.sh

+if [[ ${SPARK_RUNTIME} == 'spark2.4.8' ]]; then
+  HADOOP_VERSION=2.7.7
+  HIVE_VERSION=2.3.9
+  DERBY_VERSION=10.10.2.0
+  FLINK_VERSION=1.13.6
+  SPARK_VERSION=2.4.8
+  SPARK_HADOOP_VERSION=2.7
+  CONFLUENT_VERSION=5.5.12
+  KAFKA_CONNECT_HDFS_VERSION=10.1.13
+  IMAGE_TAG=flink1136hive239spark248
+elif [[ ${SPARK_RUNTIME} == 'spark3.0.2' ]]; then
+  HADOOP_VERSION=2.7.7
+  HIVE_VERSION=3.1.3
+  DERBY_VERSION=10.14.1.0
+  FLINK_VERSION=1.14.6
+  SPARK_VERSION=3.0.2
+  SPARK_HADOOP_VERSION=2.7
+  CONFLUENT_VERSION=5.5.12
+  KAFKA_CONNECT_HDFS_VERSION=10.1.13
+  IMAGE_TAG=flink1146hive313spark302
+elif [[ ${SPARK_RUNTIME} == 'spark3.1.3' ]]; then
+  HADOOP_VERSION=2.7.7
+  HIVE_VERSION=3.1.3
+  DERBY_VERSION=10.14.1.0
+  FLINK_VERSION=1.13.6
+  SPARK_VERSION=3.1.3
+  SPARK_HADOOP_VERSION=2.7
+  CONFLUENT_VERSION=5.5.12
+  KAFKA_CONNECT_HDFS_VERSION=10.1.13
+  IMAGE_TAG=flink1136hive313spark313
+elif [[ ${SPARK_RUNTIME} == 'spark3.2.3' ]]; then
+  HADOOP_VERSION=2.7.7
+  HIVE_VERSION=3.1.3
+  DERBY_VERSION=10.14.1.0
+  FLINK_VERSION=1.14.6
+  SPARK_VERSION=3.2.3
+  SPARK_HADOOP_VERSION=2.7
+  CONFLUENT_VERSION=5.5.12
+  KAFKA_CONNECT_HDFS_VERSION=10.1.13
+  IMAGE_TAG=flink1146hive313spark323


nit: older Spark versions can be removed.

yihua · 2023-07-25T19:01:59Z

pom.xml

    <flink.connector.kafka.artifactId>flink-connector-kafka</flink.connector.kafka.artifactId>
    <flink.hadoop.compatibility.artifactId>flink-hadoop-compatibility_2.12</flink.hadoop.compatibility.artifactId>
-    <rocksdbjni.version>5.17.2</rocksdbjni.version>
+    <rocksdbjni.version>7.5.3</rocksdbjni.version>


After reading the code, RocksDbBasedFileSystemView is the only place using RocksDb. So this should be ok.

yihua · 2023-07-25T19:05:31Z

CI has passed for 6b33d37. No need to rerun CI again.

CTTY changed the title ~~[DNM] Skip tests for Java 17~~ [HUDI-6509] Skip tests for Java 17 Jul 10, 2023

CTTY changed the title ~~[HUDI-6509] Skip tests for Java 17~~ [HUDI-6509] Add GitHub CI for Java 17 Jul 10, 2023

CTTY force-pushed the ctty/master-skiptest-j17 branch from cb10175 to 8ea8e3a Compare July 12, 2023 04:41

yihua self-assigned this Jul 13, 2023

CTTY changed the title ~~[HUDI-6509] Add GitHub CI for Java 17~~ [DNM] [HUDI-6509] Add GitHub CI for Java 17 Jul 15, 2023

CTTY changed the title ~~[DNM] [HUDI-6509] Add GitHub CI for Java 17~~ [HUDI-6509] Add GitHub CI for Java 17 Jul 16, 2023

CTTY mentioned this pull request Jul 16, 2023

[DNM] Testing Java 17 #8949

Closed

4 tasks

CTTY force-pushed the ctty/master-skiptest-j17 branch from a038fd6 to c79f37c Compare July 16, 2023 06:12

Add Hadoop conf to HiveConf for HiveSyncConfig

68f6b69

yihua added dependencies Dependency updates priority:blocker Production down; release blocker release-0.14.0 labels Jul 18, 2023

yihua reviewed Jul 18, 2023

View reviewed changes

CTTY added 14 commits July 23, 2023 23:29

Add GitHub CI for spark33/34 Java 17

10f012b

Testing docker java 17 test

b8035db

minor

e083529

test

7a86dee

All docker java17 tests passed, adding all other CIs back

6bdcdac

minor

5a07bf7

trigger github ci

e8cd0e2

fix java17 test flag

8d3cca1

Rebase onto master. Migrate one new test

44d10bd

stablize TestHoodieWrapperFileSystem

100b397

trigger ci

60eb622

add ps in alpine image

8d76f7d

minor

3f8c949

minor

26777da

CTTY added 17 commits July 23, 2023 23:29

fix path

de24436

fixing ssh command

323c1de

only test for spark 3.4

6a86262

trigger ci again

8e54816

Have 6 datanodes

5a0ed39

10 datanodes

10f4cb2

have 8 datanodes

cdaff5f

5 nodes

409e7d8

3 nodes

172b237

run java 17 ci first

d018c71

adjust ci

9bfcf7c

minor

150f0be

run clean test

d496aed

seperate CIs

91a5dec

Fix core site

c8540ca

Fix core site

3723d50

fix core-site

9bc5072

CTTY force-pushed the ctty/master-skiptest-j17 branch from 95b9ca4 to 9bc5072 Compare July 24, 2023 06:29

CTTY added 2 commits July 24, 2023 11:05

try removing spark-streaming test dep

91c5a05

add spark streaming back

6b33d37

Revert "Add Hadoop conf to HiveConf for HiveSyncConfig"

8b62b65

This reverts commit 68f6b69.

yihua approved these changes Jul 25, 2023

View reviewed changes

yihua merged commit 03bc554 into apache:master Jul 25, 2023

CTTY deleted the ctty/master-skiptest-j17 branch July 25, 2023 19:15

jonvex mentioned this pull request Nov 7, 2023

[MINOR] Remove rocksdb version from m1 profile #10006

Merged

4 tasks

mzheng-plaid mentioned this pull request Nov 5, 2024

[SUPPORT] Current state of Java 17 support #12207

Closed

hudi-bot mentioned this pull request Dec 9, 2025

Java 17 runtime support #17236

Open

rangareddy mentioned this pull request Dec 9, 2025

Remove unnecessary usage of rocksdb usage in CI tests #17249

Closed

[HUDI-6509] Add GitHub CI for Java 17 #9136

[HUDI-6509] Add GitHub CI for Java 17 #9136

Uh oh!

Conversation

CTTY commented Jul 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

CTTY commented Jul 12, 2023

Uh oh!

CTTY commented Jul 18, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CTTY Jul 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hudi-bot commented Jul 24, 2023

CI report:

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua commented Jul 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CTTY commented Jul 6, 2023 •

edited

Loading

CTTY Jul 18, 2023 •

edited

Loading